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This guide focuses on the collection, measurement, 
and evaluation of data for programs operating under the mandate of 
Chapter 1 of the Education Consolidation and Improvement Act. 
Assessment of compensatory education performance of local education 
authorities (LEAs) is based on reporting of the following data: (1) 
basic and advanced skill scores; (2) demographic information; (3) 
student progress in regular programs; (4) continued progress in 
meeting desired outcomes; and (5) sustained effects. The guide lists 
possible additional data sources for program measurement and 
evaluation. A norm-referenced model is included. Tips on interpreting 
and using evaluation results and improving the accuracy of evaluation 
data are listed* Guidelines and sample reporting forns for a Chapter 
1 annual review are included. A discussion of data disaggregation is 
provided. Data disaggregation is defined as the process of pulling 
apart test scores and other types of information concerning student 
achievement, instructional services, and staffing patterns in order 
to explore relationships between learning and program 
characteristics. This discussion includes guidelines for curriculum 
alignment and data disaggregation. A chart listing various scoring 
systems, a brief description of each system, and the advantages and 
limitations of each system is included. Checklists for Chapter 1 
evaluation quality control and test selection are appended. (AF) 
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BIST ROPY AVAILABLE 



LEA EVALUATION 

REPORTED TO SEA 

1. Basic & Advanced Skill Scores 

2. Demographic Information 

MAINTAINED IN LEA (Results reported to SEA at least once every 3 years.) 

3. Review Progress In Regular Program 

4. Assess Continued Progress In Meeting Desired Outcomes 

5. Sustained Effects 



Basic and Advancad Skill Scoras 

1. Measure achievement in reading, mathematics, and language arts In grades 
2-12 

2. With regard to more advanced skills, assess progress of Chapter 1 
participants as measured by the 'comprehension" or equivalent score of a 
nationally normed reading test; and the 'problems and applications" or 
equivalent score of a nationally normed mathematics test 

3. In assessing achievement in language arts, use tests designed to measure 
language arts or reading. If a reading test is used, the LEA shall assess 
achievement in both basic and more advanced skills. 

4. Exclude LEP projects designed to teach English to limited English speaking 
children 

5. Measure student achievement over a period of approximately 12 months 

6. Report on either a spring-to-spring or fall-to-fall testing interval 

7. Aggregate by subject and grade for grades 2-12. 

8. Report in the common reporting scale-NCEs 



Demographic Information 

1 . Collect data on the race, age, gender, and number of children with 

handicapping conditions served by the program assisted under this chapter 
and on the number of children served by grade-level under the programs 
assisted under this chapter and annually submit such data to the Secretary. 
(P.L 100-297, Sec. 1019.) 



2. The SEA shall inform its LEAs, in advance, of the specific data that will be 
needed and how the data may be collected. (Federal Regulations, 
§200.35(b)(2).) An LEA shall provide to the SEA any data needed by the 
SEA to complete its annual performance report. (Federal Regulations, 

§200.35(c)(2}.) 



Progress In the Regular Program 

Review may be based on- . . 

* Teacher Judgments 

* Grades 

• Retention Rates 

• Other Appropriate Indicators of Success 



Desired Outcomes 

1. Desired Outcomes are an LEA's goals to Improve the educational 
opportunities of educationally deprived children to help those children- 

A. Succeed in the regular educational program 

6. Attain grade-level proficiency 

C. Improve achievement in basic and more advanced skills 

2. At a minimum, must be expressed in terms of aggregate performance 

3. May also use other indicators. 



Sustained Effects 

1. Collect information to determine whether student achievement gains are 
sustained over a period of more than 12 months. 

2. Assess performance of the same children for at least two consecutive 12- 
month periods, provided these children continue to be enrolled in schools of 
the LEA. 

3. Report information on either a spring-spring-spring or fall-fall-fall testing 
interval 




POSSIBLE ADDITIONAL DATA SOURCES 



Following are some possible sources of data to look at as you choose multiple 
measures for assessing Desired Outcomes, Progress in the Regular Program, 
Individual Student Progress, and Parental Involvement. Some tounots will be 
appropriate for some purposes but riot for others. Careful consideration should be 
given to the infomfiation each data source will contribute to your analysis. 

ELEMENTARY PROGRAMS 

End of Unit/Text Tests 

• Proficiency Tests ^ 

• State/District Criterion-Referenced Tests 

• Teacher-Made Tests 

Performance on Classroom Assignments 

• Number of Bool^s Read 

Grade Placement-Success (or Lack of Success) in Being Promoted 

• Performance on Homework Assignments 

• Level in Basal Reader 

• Mastery of Classroom Material 

• Parent Involvement in Classroom Tutoring 
Parent Involvement in Home Learning Experiences 

• Attendance at Parent Training Meetings 

• Participation in Parent/Teacher Conferences 

Number of Parent Contacts With the School-letters, notes, phone calls 



SECONDARY PROGRAMS 

• End of Unit/Text Tests 

• Proficiency Tests 

• State/District Criterion-Referenced Tests 

• Teacher-Made Tests 

• Performance on Classroom Assignments 

• Performance on Homewori< Assignments 

• Mastery of Classroom Material 

• Grade Placement-Success (or Lack of Success) in Being Promoted 

• Grade Point Average 

Accrual of Credits Toward Graduation 

• Graduation Rate 

• Parent involvement in Home Learning Experiences 
Attendance at Parent Training Meetings 
Participation In Parent/Teacher Conferences 

Number of Parent Contacts With the School-letters, notes, phone calls 



MODEL A 
NORM-REFERENCED MODEL 



TREATMENT r 



OBSERVED 

POST-TREATMENT 
PERFORMANCE 



EXPECTED 

•NO-TREATMENT 
PERFORMANCE 



OBSERVED sTREATMENT GROUP'S MEAN STANDARDIZED 

SCORE ON NORMED POSTTEST 



EXPECTED s TREATMENT GROUP'S POSTTEST 

EXPECTATION DETERMINED FROM PRETEST 
PERCENTILE 



NORMAL CURVE EQUIVALENTS 
WHY NCEs 

To be useful for the purpose of measuring the irriDact of instructional 
treatments, a metric must be: 

Accurate in rejecting relative achievement levels 

• Composed of equal-size units 

• Sensitive to small gains 

• Meaningful to the users 

in addition, if comparisons are to be made or data to be aggregated, the metric 
must be able to combine results from different test instruments. NCEs were 
developed because the most widely used types of scores are deficient in one or 
more of these respects. 



WHEN INTERPRETING EVALUATION RESULTS 



Use several comparisons; donl rely on Just one standard or comparison In 
drawing conclusions for evaluative purposes. 

Calculate the median (not as sensitive to the distortion of small numbers and 

extreme scores) and the mean to evaluate gains. 

The accuracy of an NCE gain varies depending on the number of stiidents 

on which the gain is based. Dont overinterpret gains based on small 

numbers. 

The size of NCE gain tends to be related to grade level (larger at lower 
grades) and subject area (larger for mathematics projects than reading 
projects). 

Differences in initial performance level may affect size of gain scores 



IMPROVING ACCURACY OF EVALUATION DATA 

Tracl( students to ensure the largest possible proportion of Chapter 1 
students with both pre and posttests-less than 2/3 of students served is 
considered unrepresentative 

Test no more than 2 weeks from the empirical normlng date; deviations from 
the norming date should be similar in direction and length for both pre and 
posttests 

Match the test used to your instructional program 
Choose a test sensitive enough to detect the ejects of your project 
Administer the test in accordance with the publisher's instructions 
Ensure the best possible testing conditions by: 

-eliminating distractions 

-providing a comfortable setting 

-providing a testing log for teachers to note any unusual occurrences 

-familiarizing students with Hem format 
Examine carefully unusually high or low pretest and posttest NCE gains for 
possible sources of error and unreliability 



PURPOSES OF EVALUATION 

1. Detect individual strengths and weal^nesses 

2. Detect program weal^nesses 

3. Diagnose problems 

4. PROGRAM IMPROVEMENT 

5. Placement into programs 

6. Determine extent of teaching success 

7. Reporting to parents and others 
6. Compete for awards/recognition 
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CHAPTER 1 ANNUAL REVIEW 



Each school must conduct an annual review of data In 1 and 2, and possibly 
3-6, and disseminate the Information to parents, teachers, and others. 

Must be discussed at annual review 

School-level Information 

1. Aggregate gains for all educationally deprived students in grades 2 to 12, using 
Spring to Spring or Fall to Fall data. 

• Basic skills: record total reading and/or total math scores in 
NCEs 

e Advanced skills: record reading comprehension and/or mathematics 
problem solving and application scores in NCEs 

e Compute gains and losses and find the median or mean 

2. Monitor and assessi desired oLitmrnas at all grade levels using the criteria 
established for 'substantial progress." Set up databases by school for each 
outcome in your project application. 

May be discussed at annual review 

School-level infonnation 

3. Look at student level gains and att ainment of desired outcomes and make program 
modifications for children who didn't gain. 

District-level Information 

A. Look at performance in the regular prngfgm, 

5. Conduct a sustained effects study every three years. 

6. Assess parent involvement. 
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School-Level Annual Review 

School: . Lead Person: ^ 

Review of prei>ost test scores (aggregate performance for grades 2 and tilgher). 



instructional 
Area 


NCE 
Gain* 


KWCaaMAfli^ 

Focus 


Reidlng(daslc) 






Reading (Advanced) 




Math(Biilc) 






Math (Advanced) 




Lanii.Arti 







* Gain measured - spring to spring or fan to fan 

DM «M tdwol iMdi tM aggrvoM* parfoniMnM VMl* for 
thf ln«tnictlenil m that l« tfw primary focut of tiM ^wort 
Cttaptarl Program? p~| 

Vaa I I No 
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REVIEW OF DESIRED OUTCOMES STATED 
IN CHAPl ER 1 APPUCATION 



Desired Outcomes 


% Attaining 


Outcome Mel? 
Yes/No 








2. 






3. 






4. 






5. 






6. 







Did thsschctol show -subsfmitlal progress*' Vw school Is kfentftled lo fmpmeht m Chaptof 1 pmQr^ 

low&itl meeting aR desired ouloomds? YeidrNo impnv6nmtnqulmmti(S9<x 200,38) n the 

. bQltLr6dd\ m dggwgato performance goals andmgk^ sub- 

*D6flhltkHi0f'^i4)stanmpmgmss''fr(>m the siantlal progress lowani achieving tha stated desM outcomes. 
State Plan tor Chapter 1 1mprovement 



School Identified tor program imptovement? Yes or No 



SUPPORT PAGE FOR AGGREGATE PERFORMANCE 





REAI 


DING-Bi 


IVSIC 


READING-ADVANCED 


MATH-BASIC 


MATH-ADVANCED 


LANG. ARTS 


GRADE 


N 


NCE 
Avo. 
Gain 


INot 

Makina 

Gain 


N 


NCE 
Ave. 
Gain 


«Not 

Making 

Gain 


N 


NCE 
Ave. 

Gain 


«Nol 

Makkig 

Gain 


N 


NCE 
Ave. 
Gam 


«Nol 

Mi^ina 

Gain 


N 


NCE 
Ave. 
Gain 


«Nol 

Making 

Gain 


2 
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Gradd By Qradt Support Pag« for Daslrtd Outcomes 
DMirad Outeoma # 



Grada 


Attalnad 
Outcoma 


Did Not 

Attain 

Outcoma 


Total 


N 


% 


N 


% 


X 
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Has daairad outcome 
attained? 

Yaa or No 



Comment 



DMirtd Outcoma # 



Grade 


Attained 
Outcome 


Did Not 

Attain 

Outcome 


Total 


N 


% 


N 


% 


K 
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Was deaired outcome 
attained? 

Yea or No 



Comment 
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STUDENT PROGRAM IMPROVEMENT 

Identify Students Whose Performance Shows no Improvement or Decline 
Assess Their Performance Individually in Basic and Advanced Skills 

A. Must use standardized test results 

B. Use of multiple additional measures recommended 

Conduct Thorough Assessment of Their Educational Needs and Use Results 
to Modify the Project to Serve Those Students Better. 



LOCAL CONDITIONS 

The mobility of the student population 

The extent of educational deprivation among participating children that may 
negatively affect improvement efforts 

The difficulties involved in dealing with older children in Chapter 1 programs 
in secondary schools 

Whether indicators other than improved achievement demonstrate the positive 
effects on participating children of Chapter 1 activities 

Whether a change in the review cycle under section 1019 of the Act of 
paragraph (a)(1) of (§200.38] or in the measurement instrument used or other 
measure-related phenomena has rendered results invalid or unreliable for a 
particular year. 



student Identification 

«I!kS?'^-4°/, """"^'i evaluation you are required to identify all students who have not shown 
decllnS toytard neeting desired outcome^i or whose performance show no Improvement or a 

1. List all students in the Chapter 1 program. 

2. Indicate their current grade placement (during year when data was gathered). 

3. Indicate a 1 for students not meeting goals for the first time, and a 2 if this is the second for 
more) consecutive year in which the child was identified. secona tor 

4. Place (X)»s to indicate areas in which the child did not improve. 



student Name 


GR 


Read 
Basic 


ing 
Adv. 


Math 
Basic Adv. 


Lang 
Arts 


Desired 
Outcomes 


Will 

student 

be in 

program 

next 

year? 


Is 

student 
identified 
for program 
improvement? 


Number 

OX. 

years 
in 

program 


Basic 


Adv. 


Basic 


Adv. 


1 


2 


3 


4 


5 


6 


Yes 


No 


Yes 


No 
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INTERPRETING EVALUATION RESULTS 



REQUIRED INDICATORS 

Did my program overall show 
satisfactory gains? 

Did my project reach its goal? 



How did my Chapter 1 students 
progress/succeed in the regular school 
program? 



STANDARDS 

Aggregate project gains should equal 
standard set by state at a minimum 

Project objectives stated in measurable 
temis (i.e., desired outcomes) 

LEA standards based on teacher 
judgments, grades, retention rates, or 
other appropriate indicators 



USEFUL INDICATORS 

How did my 3rd grade program do 
compared to my 3rd grade program 
last year? 

How did the 3rd grade program do 
compared to the 2nd and 4th grade 
Chapter 1 program? 

How did my Chapter 1 3rd grade 
program compare to 3rd grade 
programs across the state? 



STANDARDS 

Gains from last year's project at each 
grade 



Project gains at adjacent grade levels 



Average gains for Chapter 1 projects 
across the state for the same grade 
and subject 



IS 



USING EVALUATION RESULTS 



ADMINISTRATORS DO YOU WANT 
TO... 

1. identify classes which may need 
remediation? 

2. find areas for currlcular or 
instructional revision? 

3. identify subject areas in need of 
revision or increased resources? 

4. identify areas of improvement or 
decline? 

5. determine in-service needs? 



6. compare achievement of your 
school and/or district with 
others? 

7. determine general achievement 
levels? 

6. }(now the general progress of 
students in your school and/or 
district? 

9. compare student progress to 
the progress of others in the 
state or nation? 

10. determine trends in student 
progress? 

11. find out which projects are most 
or least effective? 



LOOK AT . . . 



summaries of subtest performance for 
each classroom. 

dusters of test objectives on which 
performance was low. 

surhrnaries of subject performance by 
grade for the school and/or district. 

subtest scores by grade on previous 
tests in the same area. 

subtests on which school or district • 
performance was low. 

narrative information by subtest and/or 
grade level 



summaries of test performance with 
each grade by school. 

NCE gains by grade level for the 
school and/or district. 



typical NCE gains made by other 
schools in the state or nation. 



NCE gains compared to previous year. 



NCE gains for each project at each 
grade level. 
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DATA DISAGGREGATION' 



Data disaggregation Is the process of "pulling apart" test scores and other 
types of information concerning student achievement, Instructional services, and 
staffing patterns in order to (explore the relationships that exist between teaming and 
program characteristics. It Is one possible means of determining where to focus 
improvement efforts once a district has been Identified for program improvement 
based on aggregate performance measures. Data disaggregation must be 
preceded by careful alignment of curriculum objectives with objectives covered by 
the norm-referenced test used to evaluate the Chapter 1 program. Following are 
steps to take In determining curriculum alignment and disaggregating data. 

Curriculum Alignment 

The process of determining cuniculum alignment and subsequently using test 
scores to determine areas of weakness within the curriculum is a time-consuming 
process that requires the cooperation of staff across programs. Therefore, district 
administration must be willing to support staff members in this effort. In addition, 
the district cuniculum must be articulated In some written fomi, such as -school- 
based objectives, district-wide objectives (strategies or skill dusters), state syllabi, or 
some other type of defined format. 

Cuniculum Alignment Is a four step process, as follows: 

1. Form a school-wide committee for each content area of Interest. Both 
the Chapter 1 and the regular school program should be represented on the 
committee, along with any school-wide or district cuniculum and/or evaluation 
specialists. 

2. Develop two lists of objectives. The committee should obtain lists of 
objectives or defined skills that are taught In each program and/or content 
area of interest. For purposes of curriculum alignment between the regular 
and Chapter 1 programs, the committee would need to obtain a list of 
objectives (strategies or sk^.i clusters) for both programs in each content area 
being considered. If such lists do not exist, the committee will need to 
develop them. Each content area list should Include 1) a summary statement 
for each objective, strategy, or skill. 2) some type of code number or 
reference number for each objective, strategy, or skill, and 3) nn Indication of 
approximaieiy when each objective, strategy, or skill is taught in each 
program. This Infomiation is critical In the process of data disaggregation 
that follows. 



' Based on Reichman. S. L & Rayford. L. Using ibst results for curriculum 
alignment, an approach to program evaluation and Improvement. Paper presented 
at the Annual Meeting of the American Educational Research Association. New 
Orieans, April 5-9, 1988. 
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3. Rf view the test and group related Items. A copy of the norm-referenced 
test needs to be obtained ao that committee members can review actuai test 
Hems and group reiated items by test objective (strategy or aldii cluster). Any 
such groupings done by the test developer may be used as a basis, but the 
committee should review the groupings to ensure that H agrees with how the ' 
test publisher organized the strategies or skills. If the committee does not 
agree with the publisher's Information, then it should make the requisite 
changes regarding the manner In which items are grouped. 

Each objective (strategy or skill duster) assessed by the test should also be 
assigned a code number of a different but similar form from those assigned 
to curriculum objectives. The assessment objectivess are the link between 
the regular program and Chapter 1 Instructional objectives, and therefore 
fomi the basis for reviewing curriculum alignment. 

4. Link test objectives to related program objectives. Using the list of 
regular program and Chapter 1 objectives, along with the listing of test 
objectives and their respective code numbers, the committee should link each 
test objective to the related program objective by writing the appropriate test 
objective code number next to each program objective. This will likely 
require considerable committee discussion and perhaps some compromise. 
The results of this step indicate the congruence between objectives 
assessed on the test and the objectives taught in both the regular and 
Chapter 1 programs. 

Data DIsaQQreaatlon 

The process of determining curriculum alignment as outlined above should be 
completed before meaningful disaggregation of data can occur. The process is as 
follows: 

1. Establish minimum passing level for the test and for aach objective. 

Based on the objectives (strategies or skill dusters) outlined by the 
cuniculum alignment committee, establish a "minimum passing level" for the 
entire test and for each objective. The cut-off score should reflect the 
minimum standards that the committee feels students should attain in order 
to be classified as having passed the overall test end Its individual objectives. 
The passing level for the test will be used as a basis for grouping students 
into mastery/non-mastery and should be based on tota! raw scores. Similar 
criterion levels should be set to determine whether individual students 
mastered or did not master each objective (strategy or skill cluster). Raw 
scores should be used to dassify each student's success on each of the 
assessment objectives. 

2. Organize test results and disaggregate data. Once the test has been 
administered, test results for the entire group should first be scored as 
follows: 1) for each student each item is scored as either correct or 
incon-ect and the raw score indicating the total of items connect is obtained; 
2) for each student a determination of passing/not passing is made for the 
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overall test and for each objective (strategy or skill cluster). There are 
various way of combining score information to obtain information about 
various aspects of your program. For instance you may want to look at 
student achievement overall and on aach objective. You may want to 
examine building and classroom offsets or affects of various sarvioe delivery 
models. You may consider dropout and graduation rates, program recidivism, 
behavior and discipline referrals, differences related to race, gartder or 
ethnicity. You may iook at affects of various instructional techniques on 
different objectives or strategies. For each question of this nature that you 
wish to examine, students' scores will be grouped differently ano differences 
between groups examined. 

Interpret the rasults In view of the programs of Inttrtst. Rnally, the 
committee reviews the results of the disaggregation process to determine 
how well students are doing. As the committee reviews data. Its task is to 
identify objectives (strategies or skill dusters) where there are perfonnance 
discrepancies between groups oi studerits; for instance, do students 
demonstrate varying levels of proficiency as a function of the type of 
educational services they receive? Or. does their test perfonnance relate 
directly to the building(s) in which they receive those services? 

As is true with most test data used for program evaluation, the 
disaggregation process will highlight areas which shouid be investigated more 
closely, but will not provide specific information on potential solutions to 
problems that appear to exist. For example, results of the disaggregation 
process may indicate that a disproportionate number of students in the 
Chapter 1 program are not mastering certain reading comprehension 
strategies, even though the Chapter 1 objectives indicate that these strategies 
were taught to students prior to the testing date. A potential problem is 
highlighted by the disaggregation process, but the committee would have to 
delve further to determine why this is occurring. 



CHARACTERISTICS OF 



VARIOUS SCORE 



SYSTEMS 



TYPE OF SCORE BRIEF DESCRIPTION 80UE ADVANTAGES 

RAW SCORES Th« numbtf of oorrtct Could bo UMd to 

mponsoo obouood by assoos maoiofy in a 

t>t ttudont 0^ OTM 
Rangt: 0-Total numbor 
of tost horns. 



PERCENTILES Tho studonft roiotivo 

ttonding In tf)o norm 
group In tonm of tho 
proportion of ttudtntt 
•coring botow him^tr . 
Rango: 1*M 



Can bo UMd to 
attablifth tio standing of 
an Mr^tdual rslativo to 
a normativo group 



GRADE EQUIVALENTS -Grado Plaoomonr of Apparant aasa of 

atudont based upon tost aommunkation (Paopio 

porformanoo. b s i t vs iwy imdorstond 

Hango: K*12 whtf h maam by Qiads 

Equtvalants) 



STANDARD SCORES 
SYSTEMS 



8TANINES 



Tho amount by arhich a 
stodonrs soort daparts 
torn ^rago. 
Rar^o: dspondi on 
parliGuiar sjrstsm (o.g. 
T-aoofo; Z-acoros; 
staninas. NCEs) 

A standard soort 
systom «Mch has nina 
intorvals 



Por1om>anoas of various 
tosts and subtosti can 
bo oomputod* 

Tboy art aquat 
intorvab. 



Samo as standvd 
aoora aystoms 



EXPANDED SCALE 
SCORES 



A standard soors 
syitom with an arbitrary 
mid point but gonorslly 
inckidts savaral grado 
lovols. 

Rar>gt: Dopsnds on 
particular systom by tost 
publishor 



Samo as standard 
aoora aystoms. plus: 

Comparison can bo 
mado from grads to 
grado 

FacKutos out ot tovel 
totting 



NORMAL CURVE 
EQUIVALENTS (NCEs) 



A stondard sooro 
systom having Og 
intonrais. Tho avarago 
oonaiponds to tho 50th 
poroontils; tho 1st and 
09th NCEs aorraapond 
to tho 1st and ggth 
poroantilai 



Samo as standard 
aoora systoms. plus: 

Easo of intocprttation 

Parmit aggragation of 
dato from a wids varisty 
of tosto 



SOME UMITATIONS 

ONfiouhy in comparing 
paclorniaiiuo on 
aubpodions of tho samo 
iMks 

OWouly In comparing 
portomianoo on dftoront 



OMouliy in Imorproting 
aoocas 

Cannot bo used in 

computation of group 
atatistios. 

A dtftsranoo of 10 
poroanilos at ono point 
in tm rango is not 
oqulvaiont to a dWoront 
oMOporoontilos at 
anothar point on tho 
lango. 

Samo as poroontios . 

plus: 

Thoy ars often 
arisioadtf^g (o.g. a soors 
ot 7.0 by a f tih grado 
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intorpretation by lay* 
person 

They art latatively new 

Ttiey decertd upon 
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parcaneies 
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QUALITY CONTROL CHECKUST 
FOR CHAPTER 1 EVALUATION 

RtprtMntatlvtnm of Evaluation Rndings 

All Chapter 1 participants, except those lost through attrition, are 
included in the evaluation pro- and post-testing. 

The evaluation plan includes procedures that should minimize the 
discrepancy between the number of persons served by the project and 
the number of persons for whom both pre- and post-test achievement 
scores are available. 

a. A roster of partidpating Chapter 1 students is kept by the 

program evaiuator or designee. 

b. Procedures exist for notifying the program evaiuator when 

Chapter 1 participants change classes, schools, etc., or 
move; reasons for the change: and the student's new 
location. 

c. There is a plan for conducting make-up tests within the 

"acceptable" dates for the test(s) used. 

d. The program evaiuator is provided with reasons for 

missing data for each participant lacking pre- or post-test 
data. 



Raliabllity and Validity of Evaluation 
Instrumtnts and Proeaduras 

The instrument used for Chapter 1 evaluation has been reviewed in 
the last 3 years to ensure that it is appropriately matched to the 
cumculum of the program. 

a. This process is documented. 

b. The process was done by a committee including at least 

one Chapter 1 teacher, the Chapter 1 coordinator, a 
district administrator, and others as appropriate. 

c. The instrument was chosen based on a match between 

objectives covered by the test and instructional objectives 
of the project. 



The edition of the test is either the cun'ent or immediate previous 
edition. 



Scoring of the test will be done In-house. 

a. The answer sheets have been spot-checked to ensure 

that students filled ihem in correctly and that, if 
necessary, they can be accurately read by the scanner. 

b. Whether tests are hand- or machine-scored, someone 

other that the person who initially did the scoring checks 
a sample of the tests for accuracy in scoring. 

c. Conversion of raw scores to standard scores using 

conversion tables provided by the publisher is double 
checked for accuracy. 

d. Any computer data entry is checked for errors and results 

from computer calculations are verified. 

'"Average NCE gains for each subject and grade are computed and 
verified for each building to ensure accuracy in data reported to the 
SEA. 

Aggregate NCE gains for each subject in each building are computed 
and verified for use in the Local Annual Review of program 
effectiveness. 



Valid Interpretation and Use of Results 

Project staff understand the Chapter 1 metric (the NCE) used to 
measure Chapter 1 gains. 

Evaluation results are made available to project staff for review. 

Project staff use evaluation results to ensure the quality of testing and 
evaluation procedures AND to guide improvements in overall program 
quality. 

Project staff examine evaluation results for information about student 
needs at various grade levels and in different subject areas. 

Chapter 1 and regular classroom teachers review data diagnostlcally to 
plan instruction for Chapter 1 students. 

Parents and/or students have access to evaluation results and are 
provided with assistance in understanding them. 



If out-oMevel testing is necdssary and machine-scoring is planned, 
does the scoring service provide a dear description of what information 
is required In order to obtain correct oonverslons? 

Will it be possible to obtain pretest and posttest scores for at least two-thirds 
of the students in the Chapter 1 program? 

Will appropriate test administration and scoring procedi*es be followed? 

Are the instructions clear for those (teachers/aides) who will administer 
the test? 

Is the length of time required to administer the test acceptable? 

Is the test group or individually administered, and which do you prefer? 

Will testing conditions be supportive of the students' best efforts during 
the pretest and the posttest? 

Will tests be scored accurately? 

Will average standard scores be calculated correctly only for those 
students with both pretest and posttest scores? 

Will average standard scores be converted correctly to oercentiles and 
NCEs? 

Will the scoring procedure provide timely results? 
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CHECKLIST FOR CHOOSING A TEST FOR CHAPTER 1 

Does the content of the test match the content of the Chapter 1 program? 

Are most of the Chapter 1 program's Instructional objectives r jasured ' 
by the test Items? 

Are most of the test's objectives taught in the Chapter 1 program ' 

Are the results reported in terms of the objectives? 
Is the test appropriate for the students? 

Will the students find the layout of the test booklet easy to follow? 

Is the reading level of the test appropriate? 

Will the students be able to understand the Ir^ructions? 

Is the response form simple enough? 

Does the test have representative, empirical norms? 

Does the nomi group Include students like the students in the Chapter 
1 program? 

Does the nomfi group Include school systems likely to have the regular 
instructional programs like your school system in the appropriate 
grades? 

Have you examined the most recent or immediately preceding edition 
of the test? 

Does the test have empirical norming dates that are within two weeks of 
when the pretest and posttest are to be administered? 

Will at least two-thirds of the program occur between pretesting and 
posttesting? 

Does the test have a level which matches the functional level of the students 
in your Chapter 1 program? 

What is the functional level of your students? 

If out-of-level testing is necessary and hand-scoring Is planned, does 
the person responsible for obtaining average standard scores and 
percentiles understand the proper conversion procedures? 



